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ABSTRACT 

To document the effectiveness of instruction, a retrospective 
pre-post type of evaluation was used to evaluate the effectiveness of a 
continuing education program designed for adults involved in community 
economic development. At the conclusion of the learning experience, the 35 
participants were asked to complete a brief questionnaire that included four 
demographic items and nine statements regarding program content. An analysis 
of participant responses revealed that the educational program increased 
their knowledge level with respect to each of the statements. Participants 
increased their understanding of the components of strategic planning as a 
result of the small group work, electronic slides, and written materials of 
the program. (SLD) 
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Abstract 

Documenting program impact is of primary importance for educators of all types in today’s 
political economy. To better document the effectiveness of instruction, a retrospective pre-post 
type evaluation was used to evaluate the effectivness of a continuing education program 
designed for adults involved in community economic development. At the conclusion of the 
learning experience, participants were asked to complete a brief questionnaire that included four 
demographic items and nine statements regarding the program’s content such as: “I have a basic 
awareness of the mechanics of strategic planning”. An analysis of participant responses revealed 
that the educational program increased their level of knowledge with respect to each of the 
statements. 
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Using A Retrospective Pre-Post Questionnaire To Determine Program Impact 
Introduction 

Documenting program impact is important for all educators in today’s political economy. 
The use of retrospective pretest program evaluation provides educators an opportunity to see 
students’ perceptions of their change in knowledge, skills, attitudes, and behavior after 
participation in an educational intervention. Moreover, this self-reported information not only 
enables the instructor to measure participants’ change, it also permits improved documentation 
of impact of the instruction. 

To better document the effectiveness of instruction, a retrospective pretest was used with 
thirty-five participants of a continuing education program designed for adults involved in 
community economic development. At the conclusion of the learning experience, participants 
were asked to complete a brief questionnaire that included: [l]four demographic items; and [2] 
nine statements regarding the program’s content such as: “I have a basic awareness of the 
mechanics of strategic planning”. Results indicated that the participants increased their 
understanding of the components of strategic planning as a result of the small group work, 
electronic slides, and written materials. 

Extension Education as a Way of Teaching and Learning 

Extension, as an arm of the land grant university, has been involved in informal education 
at the local community-level since early in the 20 th century. With the passage of the Smith-Lever 
Act in 1914, the idea of placing Extension educators in every county to work with local 
committees, organizations, and residents in conducting programming to address local needs 
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became a reality. Through the relationship with the local agent, community residents have had 
access to a variety of resources, research-based information, and campus-based personnel for 
nearly a century. 

The well-defined programming areas within Extension today were not evident in its early 
years. In the beginning, one agent was to work in the areas of agriculture, home economics, and 
with other issues of community concern. As the informal educational network evolved, 
however, distinctions began to emerge which prioritized educational programming into four 
areas: agriculture and natural resources, community development, family and consumer 
sciences, and 4-H-youth development (Conglose, 2000). 

As the communities with which Extension works have evolved, the relationship that 
exists among these communities, the university, and Extension has also evolved. What began as 
a county agent working on a variety of projects across what we think of today as the four 
program areas, now most often involves several full time agents in one county focusing on many 
different community issues. 

The educational model used in Extension outreach is unique in that it involves 
communities, stakeholders, and universities in ongoing conversations to define issues and 
problems on which educational programming can focus. In 2002, Peters referred to this as 
“practical public work” and stated that this was the predominant mode of Extension outreach for 
the first fifty years of Extension’s history. 

A key component of this “practical public work” involves an active local constituency 
which is engaged with the local agent in “planning and developing programs, non-formal 
teaching, facilitating meetings and community forums, providing technical expertise, and 
applying research-based knowledge to the problems of individuals, families, businesses, and 
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communities” (Peters, 2002). While it can be argued that with the technological advancements 
of the recent past, Extension outreach efforts have become more focused on technology transfer, 
this “practical public work”, which characterized Extension outreach efforts in the first part of 
the 20 th century, continues to be the dominant thrust of Extension’s community development 
efforts. 

Peters (2002) described three distinctive types of learning that are made possible through 
this philosophy of educational outreach: instrumental, communicative, and emancipatory. 
(Habermas, 1971; Cranton, 1998; in Peters, 2002). Engaging local residents in programming 
designed to enhance their understanding of how tax incentive programs work, or how to go about 
forming a community improvement corporation, or how to develop a tourism and visitors’ 
bureau would all be examples of instrumental learning. 

Communicative learning takes place when one is involved in activities or exercises that 
lead to better understanding of “each other's views, problems, hopes, and interests.” Last, when 
members of a community are engaged in programming that enhances their “leadership, 
confidence, and courage and enable[s] them to act together to change the world in ways that 
furthers] their values and ideals,” they have experienced emancipatory learning (Peters, 2002). 

Much of the educational effort conducted in Extension’s community development 
programming area yields these types of learning. However, often the challenge is being able to 
accurately measure the impact of these educational efforts. 

The Retrospective Pretest 

Documenting changes in knowledge and behavior as a result of Extension programming 
can be done simply and efficiently using the retrospective pre- post-test evaluation (Rockwell, 
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1989; Stevens, 1999). This tool “is specifically useful for evaluating the impact of Extension 
programs by asking participants to report actual changes in behavior” (Stevens, 1999). The 
retrospective pretest design, (also referred to as post- then pre- and retrospective pre- post-test 
(Rockwell, 1989; Pratt et al, 2000) is not unlike the typical pretest-posttest. The primary 
difference, however, is that it is administered only once. 

At the conclusion of some experience, program, or treatment, participants are asked to 
share the knowledge or attitude they had toward a particular subject after the experience or 
treatment. As part of the same questionnaire, participants are also asked to share the knowledge 
or attitude they had toward this subject prior to the experience or treatment. 

The retrospective pretest is similar to a more traditional pretest/posttest evaluation 
method, however, the retrospective pretest provides the participant with a “baseline” level of 
knowledge. That is, when the participant is asked to respond to a question about how much they 
know about a particular subject after they have some basic knowledge of the subject itself, they 
are more able to accurately reflect on the degree of change in knowledge or attitude (Rockwell, 
1989). Without some basic level of understanding of the topic or concepts discussed, 
participants are unable to correctly assess their baseline level of understanding. Furthermore, 
respondents will oftentimes overestimate their level of knowledge on a particular subject when 
using the traditional pretest-posttest (Pratt, McGuigan, and Katzev, 2000). With the 
retrospective pretest methodology, respondents are given an opportunity to learn how much they 
know about a subject prior to responding to a questionnaire. 

The retrospective pre-test can also be more accurate because it is answered in the same 
frame of reference as the post-test. Doing so reduces the chances that respondents score better on 
a post-test as a result of their exposure to a pretest. This is also referred to as "response-shift 
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bias" in self-report pretest/post-test designs and can be minimized through use of the 
retrospective pre-test design according to Pratt et al. (2000). 

Using the Retrospective Pretest in Extension Education 

Extension educators have long known the importance of conducting program evaluations 
and evaluations of teaching effectiveness to improve programs and teaching methods . However, 
with the passage of the Government Performance and Results Act of 1993 (GPRA), a renewed 
emphasis was placed upon the program effectiveness component in Extension. (Richardson, 
Gamble, and Mustian, 1998; O’Neil, 1998). 

The GPRA requires all federally funded agencies to develop and implement an 
accountability system based on performance measurement (Government Performance and 
Results Act of 1993 webpage). One way to address GPRA accountability issues involves the use 
of the retrospective pretest design to measure the performance of Extension programming. This 
methodology was used to determine change in knowledge, skills, and attitudes toward strategic 
planning among 35 economic development professionals after their involvement in a traditional 
Extension educational program. 

The educational program was one part of a multi-part series designed specifically for 
economic development professionals and community officials involved in local economic 
development efforts. The program, roughly two hours in length, was held in a formal classroom- 
type setting. Instructional methods involved powerpoint slides, small group discussion, and role 
playing. Participants expressed a relatively high level of interest in the topic, as many were 
previously involved in strategic planning exercises. 




9 

Following the presentation and related group activities, participants were asked to 
complete a short questionnaire. This questionnaire was part of the informational materials 
provided to each participant and was identified by a different color paper and layout (eg. font, 
landscape). 

Design 

The two-sided, one page questionnaire was formatted in the landscape orientation. The 
front page of the questionnaire contained four background questions designed to collect basic 
personalogical data. Specifically, information was requested on the role played in economic 
development; the number of years of experience in these roles; the population of the community 
on which these efforts are focused; and, how often their organization engages in a formal 
strategic planning process with its members and clientele. The reverse side contained the 
retrospective pretest. 

The retrospective pretest was designed with instructions at the top, an example, and nine 
statements. The statements were developed using the learning objectives for the strategic 
planning workshop. Participants were asked to indicate their level of agreement with each of the 
statements before and after the workshop using a six-point, Likert-type scale; (1 - strongly 
disagree and 6 - strongly agree). 

Administration 

Workshop participants were asked to complete the one page questionnaire at the 
conclusion of the program. A conscious attempt was made by the instructor to downplay the 
instrument and there was no verbal instruction provided for completing the two-part 
questionnaire. Participants were simply asked to place their completed questionnaire on a table 
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at the back of the room as they exited. Of 35 workshop participants, 32 questionnaires were 
completed. One of the 32 submitted was only partially completed (front side only). 

Data input/analysis 

The questionnaire data was analyzed using SPSS 10.1, to determine if participation in the 
workshop affected participant knowledge, awareness, confidence, and attitude. While the SPSS 
software is quite capable of examining the degree of change (among numerous other data 
analysis procedures) the degree of change was not examined. Group means (before and after) 
were also examined. 

Results and Discussion 

The retrospective pretest indicated that workshop participants experienced a positive 
change in knowledge, awareness, confidence, and attitudes, in general. There were positive 
changes in all nine workshop indicators. All but one of the nine workshop indicators registered 
positive change for at least one third of the respondents. Furthermore, the overall mean for the 
nine items increased from 3.9 (before) to 4.9 (after) (see Table 1). 

Table I 

Paired t-Tests for Retrospective Pretest fn=31) 

Variable 

I have a basic awareness of the mechanics of strategic 
planning. 



I think I could facilitate a strategic planning process. 





Mean 


sd. 


Pre-test 


3.7 


1.6 


Post Test 


4.7 


1.0 


Pre-test 


3.5 


1.4 


Post Test 


4.7 


1.1 


Pre-test 


3.3 


1.5 


Post Test 


4.5 


1.1 
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I have the skills necessary to facilitate a strategic planning 


Pre-test 


3.5 


1.4 


process. 


Post Test 


4.4 


1.2 


Strategic planning can provide direction to an 


Pre-test 


4.4 


1.6 


organization’s efforts. 


Post Test 


5.2 


1.1 


I would like to try facilitating a strategic planning process 


Pre-test 


3.6 


1.6 


at some point. 


Post Test 


4.6 


1.3 


I will attempt some form of strategic planning process in 


Pre-test 


4.0 


1.8 


the future. 


Post Test 


5.0 


1.1 




Pre-test 


4.6 


1.6 


Thinking strategically is a worthwhile practice. 


Post Test 


5.4 


1.0 


Strategic planning is an ideal way to guide an 


Pre-test 


4.4 


1.7 


organization’s economic development efforts. 


Post Test 


5.3 


1.0 



Note. All items were significant at .05 alpha. N=31. 



After participating in the workshop, nearly three-quarters of respondents registered an 
increased level of confidence in their abilities to facilitate a strategic planning process. 
Corresponding to this increased level of confidence was a change (+62%) in understanding the 
skills necessary to facilitate a strategic planning process. Nearly that same amount (57%) 
registered a change in the desire to try facilitating a strategic planning process at some point. 

With regards to recognizing the key components of strategic planning, only 1 5% 
registered a change as a result of their participation in the workshop. Taking into account the 
workshop participants’ exposure to such processes through their place of employment could 
explain the relatively low degree of change in knowledge (Note: one half of those responding 
indicated their employer engages in a formal strategic planning process annually.) 
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Group means (before and after) for each of the indicators showed an increase in 
knowledge, awareness, confidence, and attitudes. The following three indicators had the greatest 
increase: “I know what the key components of strategic planning are”, “I think I could facilitate 
a strategic planning process”, and “Strategic planning is an ideal way to guide an organization’s 
economic development efforts”. . 

Conclusions and Recommendations 

Extension educators and others looking for simple and effective methods of improving 
program evaluation may find the retrospective pretest a useful tool. The simple retrospective 
pretest used as part of this strategic planning workshop demonstrated a new and different way to 
evaluate the impact of this type of Extension programming. Simple statistical analysis illustrated 
a change in respondents as a result of their participation in the workshop. 

However, prior to using this methodology for program evaluation, there are a few things 
that require attention: A two-part (pre and post) questionnaire must be developed that includes 
the program objectives. Armed with the most basic electronic word processor skills, one can 
develop a questionnaire with little trouble. Once a template has been developed, modifying a 
questionnaire for different programs could be easily done . 

An understanding of how the data collected will be analyzed is also helpful. Program 
evaluations that involve a relatively large number of questions and/or respondents will require 
the use of the computer for data analysis. For someone without a great deal of experience using 
data analysis software, setting up this data file could be a daunting task. However, once 
established, data can be entered with little instruction and the analyses described in this paper can 
be done with little trouble. 
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This method of determining impact may take more time than many of us have been 
accustomed to devoting to program evaluation. Nevertheless, as more emphasis is being placed 
on documenting the impact of our work, this program evaluation method is relatively quick to 
develop, easy to administer, and provides much more meaningful information than many of the 
methods currently being used. 
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